Sparse LS-SVMs using additive regularization with a penalized validation criterion

This paper is based on a new way for determining the regularization trade-off in least squares support vector machines (LS-SVMs) via a mechanism of additive regularization which has been recently introduced in [6]. This framework enables computational fusion of training and validation levels and allows to train the model together with finding the regularization constants by solving a single linear system at once. In this paper we show that this framework allows to consider a penalized validation criterion that leads to sparse LS-SVMs. The model, regularization constants and sparseness follow from a convex quadratic program in this case. Regularization has a rich history which dates back to the theory of inverse ill-posed and ill-conditioned problems [12]. Regularized cost functions have been considered e.g. in splines, multilayer perceptrons, regularization networks [7], support vector machines (SVM) and related methods (see e.g. [5]). SVM [13] is a powerful methodology for solving problems in nonlinear classification, function estimation and density estimation which has also led to many other recent developments in kernel based learning methods in general [8]. SVMs have been introduced within the context of statistical learning theory and structural risk minimization. In the methods one solves convex optimization problems, typically quadratic programs. Least Squares Support Vector Machines (LS-SVMs) [9, 10] are reformulations to standard SVMs which lead to solving linear KKT systems for classification tasks as well as regression and primaldual LS-SVM formulations have been given for kFDA, kPCA, kCCA, kPLS, recurrent networks and control [10]. The relative importance between the smoothness of the solution and the norm of the residuals in the cost function involves a tuning parameter, usually called the regularization constant. The determination of regularization constants is important in order to achieve good generalization performance with the trained model and is an important problem in statistics and learning theory [5, 8, 11]. Several model selection criteria have been proposed in literature to tune the model to the data. In this paper, the performance on an independent validation dataset is considered. The optimization of the regularization constant in LS-SVMs with respect to this criterion proves to be non-convex in general. In order to overcome this difficulty, a reparameterization of the regularization trade-off has been recently introduced in ESANN'2004 proceedings European Symposium on Artificial Neural Networks Bruges (Belgium), 28-30 April 2004, d-side publi., ISBN 2-930307-04-8, pp. 435-440